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VERIFYING RESULTS OF AUTOMATIC IMAGE RECOGNITION 

FIELD OF THE INVENTION 

The present invention relates generally to 
computerized image recognition systems, and specifically 
5 to methods and systems for enabling human operators to 
verify results in such systems. 

BACKGROUND OF THE INVENTION 

There are many methods known in the art for enabling 
Q human operators to verify results of computerized optical 

fS 10 character recognition (OCR) . These methods have arisen 
^ out of the need for very high accuracy in coding of 

ffj textual and numeric characters, particularly in the area 

^ of document processing. For example, when checks are 

s processed for clearing by a bank, errors in reading the 

zz 15 amount of the check can be very expensive. Because 

y ; 

M verification by human operators is typically the most 

12 costly step in document processing, as well as one of the 

M least reliable steps, techniques have been developed for 

facilitating this step. 
20 U.S. Patent 5,455,875, whose disclosure is 

incorporated herein by reference, describes a system and 
method for correction of OCR with display of image 
segments according to character data. The method is 
implemented in document processing systems produced by 
25 IBM Corporation (Armonk, New York) , in which the method 
is referred to as "SmartKey." The system presents to the 
human operator a "carpet" of character images on the 
screen of a computer terminal. The character images, 
each containing a single character, are produced by 
30 segmenting the original document images that were 
processed by OCR. Segmented characters from multiple 
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documents are sorted according to the codes assigned to 
them by the OCR. The character images are then grouped 
and presented in the carpet for verification according to 
their assigned code. 
5 For example, the operator might be presented with a 

carpet of characters that the OCR has identified as 
representing the letter "a." Under these conditions, it 
is relatively easy for the operator to visually identify 
OCR errors, such as a handwritten "o" that was 
10 erroneously identified as an "a." The operator marks 
Q erroneous characters by clicking on them with a mouse. 

qJ Thus, displaying the composite, "carpet" images to the 

N operator, made up entirely of characters which have been 

ny recognized by the OCR logic as being of the same type, 

15 enables the operator to rapidly recognize and mark errors 
on an exception basis. Once recognized, these errors can 
then either be corrected immediately or sent to another 
M operator for correction. The remaining, unmarked 

\M characters in the carpet are considered to have been 

Li . 20 verified. 

Because of the ubiquity of OCR applications, far 
more research and development effort has been invested in 
OCR (including OCR verification) than in other branches 
of computerized image recognition that do not deal 
25 exclusively with characters. In the context of the 
present patent application and in the claims, the term 
"character" is used in its conventional sense, to refer 
to a symbol that - serves as an atomic unit of 
representation in a written language or numerical system. 
30 Characters are atomic in the sense that they cannot be 
divided into smaller sub-units without losing their 
linguistic or numerical meaning. Thus, characters that 
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are segmented, recognized and verified in OCR systems are 
generally individual letters and digits, although they 
may also be atomic representations of complex sounds, as 
in Chinese or Japanese. On the other hand, the inventors 
5 are unaware of any publications suggesting methods or 
systems for efficient verification of non-character 
computer image recognition results. 

SUMMARY OF THE INVENTION 
Preferred embodiments of the present invention 
O 10 provide an efficient and reliable method for verifying 
r? results of automated image recognition for applications 

SI in which the image features that are recognized are not 

JJ- individual characters in a language or numerical system. 

*0 After computer analysis has identified certain image 

15 elements in a group of images (or possibly in a single 
3 large image) , a number of the elements that were assigned 

^ the same classification are displayed together for a 

y human operator. The elements are typically selected and 

2 cropped from different locations in the images. They are 

20 preferably displayed together for the operator in a grid 
pattern on a computer screen, as in the above-mentioned 
SmartKey system. The operator can then verify that all 
of the elements were correctly classified and, if 
necessary, can indicate to the computer which 
25 classifications may be erroneous, typically by using a 
pointing device, such as a mouse, to select the 
incorrectly-identified elements in the grid display. 

The present invention thus extends the advantages of 
accurate and efficient verification of image recognition 
30 results to a broad range of applications beyond the field 
of OCR. Applications that may benefit from the present 
invention include, for example, computer recognition of 
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words, of non-character symbols and of features of 
three-dimensional objects. Other applications will be 
apparent to those skilled in the art. Although preferred 
embodiments are described herein with reference to 
verifying results of image analysis performed 
automatically by a computer, the principles of the 
present invention can similarly be applied to verifying 
results of image feature recognition performed by human 
operators . 

There is therefore provided, in accordance with a 
preferred embodiment of the present invention, a method 
for image processing, including: 

analyzing one or more images so as to determine a 
respective classification for each of a multiplicity of 
elements in the images, wherein the elements are not 
individual characters in a language or numerical system; 

displaying together for a human operator a plurality 
of the elements that have the same classification and 
were found at different locations in the one or more 
images; and 

receiving an input from the operator indicative of 
whether the computer erred in the classification of any 
of the displayed elements. 

In a preferred embodiment, the elements include 
pictures of three-dimensional image features. In another 
preferred embodiment, the elements include words of more 
than one character. In still another preferred 

embodiment, the elements include non-alphanumeric 
symbols . 

Typically, analyzing the one or more images includes 
carrying out a process of automated image analysis using 
a computer. 
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Preferably, displaying the plurality of the elements 
includes dividing the one or more images into segments, 
such that one of the plurality of the elements is 
contained in each of the segments, and displaying the 
segments containing the elements. Most preferably, 

displaying the segments includes displaying the segments 
in a grid pattern on a computer display. 

Further preferably, displaying the segments includes 
displaying the segments on a computer display, and 
receiving the input includes sensing a selection of one 
of the plurality of the elements on the computer display, 
wherein the selection is made by the operator using a 
pointing device associated with the computer. Typically, 
the selection of the one of the elements indicates that 
the classification of the element is erroneous. In a 
preferred embodiment, the operator is prompted to correct 
the erroneous classification. 

There is also provided, in accordance with a 
preferred embodiment of the present invention, apparatus 
for image processing, including a verification terminal, 
which is arranged to verify results of analyzing one or 
more images so as to determine a respective 
classification for each of a multiplicity of elements in 
the images, wherein the elements are not individual 
characters in a language or numerical system, by 
displaying together for a human operator a plurality of 
the elements that have the same classification and were 
found at different locations in the one or more images, 
and receiving an input from the operator indicative of 
whether the computer erred in the classification of any 
of the displayed elements. 
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Preferably, the apparatus includes a display screen, 
which is driven by the terminal to display the segments, 
and a pointing device, which is coupled to the terminal 
so as to be used by the operator to select one of the 
plurality of the elements on the computer display. 

There is additionally provided, in accordance with a 
preferred embodiment of the present invention, a computer 
software product, including a computer-readable medium in 
which program instructions are stored, which 
instructions, when read by a computer, cause the computer 
to verify results of analyzing one or more images so as 
to determine a respective classification for each of a 
multiplicity of elements in the images, wherein the 
elements are not individual characters in a language or 
numerical system, by displaying together for a human 
operator a plurality of the elements that have the same 
classification and were found at different locations in 
the one or more images, and receiving an input from the 
operator indicative of whether the computer erred in the 
classification of any of the displayed elements. 

The present invention will be more fully understood 
from the following detailed description of the preferred 
embodiments thereof, taken together with the drawings in 
which : 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a schematic, pictorial illustration of 
apparatus for verification of computer image recognition 
results, in accordance with a preferred embodiment of the 
present invention; 

Fig. 2 is a flow chart that schematically 
illustrates a method for verification of computer image 
recognition results, in accordance with a preferred 
embodiment of the present invention; and 

Figs. 3-5 are schematic representations of a 
computer screen display presenting computer image results 
for verification, in accordance with preferred 
embodiments of the present invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

Fig. 1 is a schematic, pictorial illustration of 
apparatus 20 for verification of computer image 
recognition results, in accordance with a preferred 
embodiment of the present invention. An image capture 
device 22, typically a scanner or digital camera, 
generates an electronic image, which is processed by a 
computer to identify specified image features. The 
identified features are cropped from their original 
images and are grouped with other features that have been 
assigned the same identification. A verification 

terminal 24 displays the grouped features on a monitor 
screen 26 for verification by a human operator. The 
operator uses input devices such as a keyboard 2 8 and a 
mouse 30 to mark any incorrect identifications and, 
optionally, to correct them, as well. Terminal 24 
maintains a link between each displayed feature and 
location of the feature in the original image in which it' 
appeared, so that inputs by the operator can be linked 
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back to the original images for verification or 
correction of image recognition results. 

Terminal 24 typically comprises a general-purpose 
personal computer or other suitable computing device, 
5 which is equipped with software for carrying out the 
functions of the present invention, as described herein. 
The software may be downloaded to terminal 24 in 
electronic form, over a network, for example, or it may 
alternatively be supplied on tangible media, such as 
10 CD-ROM or DVD, for installation on the terminal. 
Alternatively, terminal 24 may comprise custom hardware 
yg elements with firmware for performing these functions. 

Fig. 2 is a flow chart that schematically 
illustrates a method for verifying image recognition 
;jS 15 results, in accordance with a preferred embodiment of the 
M present invention. At a segmentation step 40, an image 

^ processing computer (not shown) identifies elements or 

yn features of possible interest in an image or set of 

«; images. Examples of element types to which the present 

CS 20 method can be applied are shown in Figs. 3-5 and 
described hereinbelow. The computer segments the image, 
into regions of interest, typically rectangular regions, 
each containing a single one of the elements. The 
computer processes the elements, using methods of image 
25 analysis known in the art, to determine an identification 
or classification for each of the elements, at a 
classification step 42. 

In preparation for verification of the recognition 
results, the elements identified and classified in steps 
30 40 and 42 are grouped by classification, at a 
classification grouping step 44. Terminal 24 receives a 
group of such elements, sharing a common classification, 
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and displays the regions of interest containing the 
elements in a grid pattern on screen 26. This 
arrangement is similar to a SmartKey carpet of character 
images, as described in the above-mentioned U.S. Patent 
5 5,455,875, except that in preferred embodiments of the 
present invention, the image elements are not individual 
characters. An operator viewing screen 26 is informed of 
the common classification and selects the elements that 
do not fit the classification, at a user selection step 
10 46. Preferably, the operator identifies the 

^ incorrectly-classified elements for terminal 24 by 

gg clicking on them with mouse 30. 

fir. 

r\ When the operator has finished selecting the 

On incorrect elements (or when there are no incorrect 

15 elements on the screen) , he or she indicates to the 
M terminal that verification of this screen is completed, 

IU typically by clicking on a "DONE" button on screen 2 6 or 

SH pressing a key, such as the "ENTER" key, on keyboard 28. 

r; Any elements on the screen that have not been selected by 

Q 20 the operator as erroneous are marked by terminal 24 as 
r ™ having been verified. Optionally, the operator enters 

the correct classification of the incorrectly-classified 
elements, at a correction step 48. Alternatively, the 
correction may be carried out by a different operator, 
25 who typically views the elements to be corrected in their 
original context. Terminal 24 maintains a link between 
each of the elements displayed on screen 26 and its 
original location in one of the input images, so that the 
verification and/or correction of the element can be 
30 properly associated with the original location. 

Fig. 3 is a schematic illustration of screen 26, on 
which a grid of image elements 60 is presented for 
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verification, in accordance with a preferred embodiment 
of the present invention. In this example, a group of 
electrical schematic diagrams was processed by computer 
so as to identify symbols corresponding to fifty-ohm 
5 resistors, and the results are presented on screen 26. 
An operator viewing screen 26 marks elements 62, 64 and 
66, by' clicking on them with mouse 30, as being symbols 
of other types, which were erroneously identified as 
resistors. Optionally, the operator may also verify that 

10 the computer has correctly read the numbers associated 
with each of the symbols. 

Fig. 4 is a schematic illustration of screen 26, on 
which a grid of image elements 70 is presented for 
verification, in accordance with another preferred 

15 embodiment of the present invention. In this case, the 
computer has processed an aerial reconnaissance image in 
order to identify aircraft appearing in the image. The 
operator marks elements 72 and 74 as comprising image 
features other than aircraft. Similar verification 

20 techniques may be used in other image analysis and 
inspection applications, such as identifying and checking 
the values of electrical components inserted into a 
printed circuit board. A similar type of display and 
approach can be used for verifying results of image 

25 analysis and feature identification performed by human 
operators . 

Fig. 5 a schematic illustration of screen 26, on 
which a grid of image elements 80 is presented for 
verification, in accordance with yet another preferred 
30 embodiment of the present invention. In this case, the 
computer has scanned a set of documents in order to 
locate occurrences of a given word, such as the day of 
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the week, "Sunday." An element 82, however, referring to 
an ice cream sundae, has been mistakenly classified by 
the computer. The operator marks this element for 
correction . 

It will be appreciated that the preferred 
embodiments described above are cited by way of example, 
and that the present invention is not limited to what has 
been particularly shown and described hereinabove. 
Rather, the scope of the present invention includes both 
combinations and subcombinations of the various features 
described hereinabove, as well as variations and 
modifications thereof which would occur to persons 
skilled in the art upon reading the foregoing description 
and which are not disclosed in the prior art. 
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